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ABSTRACT 

A study involving all students taJsing the California 
Achievement Test (CAT) in the Cleveland City School District (Ohio) 
during the fall of 1989 was conducted to assess the effects of the 
use of parallel test forips. Each spring, all students in grades 4, 6# 
and 8 are selected to take the appropriate level of the CAT reading 
testr Form E* As a result^ the question of test familiarity was 
raised* To address this question. Form F of the CAT battsry was 
selected for administration in the fall of 1989. The sample 
population included 4,367 fourth, 3,768 sixth, and 2,770 eighth 
graders. Data from each annual testing from the spring of 1988 
through the spring of 1990 were compared with the fall 1989 data. 
Normal curve equivalent reading scores were used to identify 
differences twtween CAT-E and CAT-F scores. To determine whether 
scores differed significantly for the CAi'-E versus the ChT-F, a 
regression discontinuity design was used to investigate the trends in 
reading achievement evident for each cohort. Results refute the 
notion that repeated use of the same form ct the test improves scores 
through '•practice effects." The alternative form of the test used 
during the saite academic year provided comparable, or higher, reading 
achievement levels when used for the first time in the school 
district. Six figures and one tcJble are included. (TJH) 
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Test Familiarity: Evidence of "Practice Effects"? 
Introduction 

The recent attention turned towards standardized 
achievement testing on a national level has included the 
issues of test design adequacy, extent of testing, the use of 
test results and their validity over the period of time the 
test is in use. Additionally, the escalation of test related 
accountability issues has initiated close scrutiny of testing 
parctices among schools and teachers including charges of 
unethical behavior in the preparation of students for the 
testing experience. 

One element of the test design involves the utilization 
of parallel forms of the test to provide an optional torm for 
use in a testing program. Another element of the parallel 
form issue involves the use of a test's alterm>te form as a 
method of preparation for a testing experience. This study 
investigates the measured achievement performance of students 
participating in a testing program which includes the use of 
both forms of a standardized test. 

Lake Woebegon and Ethical Testing Practices 



In 1987 and again in 1989, John J. Cannell published 



reports rhlch asked serious questions about the use of, 
validity of, and reliance upon standardized achievement test 
results by school districts and states throughout the nation. 
Considerai^le reaction to these reports has fostered numerous 
studies and reviews which have criticized the attacks while 
admitting that test results could be used to mislead the 
public. Highlights of the issues and practices related to 
the use of parallel forms of published achievement tests are 
noted below. 

Parallel Form Testing 

Parallel (or equivalent) forms of tests are made 
available by publishers in order to "allow teacners to retest 
students at different times to evaluate progress without 
having them retake the same test or to test students in 
different sections of a class without being concerned about 
test security" (Sax, 1974). Typically, alternate form 
reliability studies are cited by the publisher in order to 
support the notion of equivalency of measure despite the form 
being employed. Additionally, Sax (1974) notes that the use 
of alternate forms makes it difficult for a student to 
recognize items from a previously administered test or to 
help another person planning to take the same test. Although 
parallel forms serve these purposes, in practice they are 
seldom used. School districts who face significant cost 
increases to pur -ihase two sets of a test in order to 
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implement ft parallel form testing program typically employ 
alternate forms only for specialized or specific situations. 

Recent criticisms of local and state reliance upon 
standardized achievement test measures have included 
allegations of various test preparation practices utilized in 
schools. Koretz (1988) indicated that "the vast gray area of 
teaching to the test stretches from frank cheating at one 
extreme to appropriate remediation and instruction at the 
other. Both educators and educational researchers disagree 
strenuously about where the line between appropriate and 
inappropriate teaching should be drawn." In a response to 
Dr. Cannell's allegations (1987), Phillips and Finn (1988) 
cite the 

practice (among schools) of continually using the same 
test for many years within the school system. Teachers 
become familiar over time with the test objectives, and 
students get accustomed to the item formats. In some 
cases, teachers may actually teach the specific items 
on the test. These "practice effects" greatly inflate 
the students' scores bnd give a misleading impression 
of achievement gains. 

Test preparation activities were addressed by Mehrens 
and Karoiniski in 1939 whereupon they described a continuum of 
practices ranging from always ethical practices on one hand 
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to never ethical pr&ctices on the other. Seven categories 
included; 1) general instruction, 2) teaching test taking 
skills, 3) instruction by objectives not matched to the test, 

4) instruction by objectives matched to the test, 

5) instruction on matched objectives using the test format, 6) 
practice on a published parallel form of the test,, and 7) 
practice on the same test. The authors concluded that the 
point where one crosses over from legitimate to illegitimate 
practice "must be somewhere between (3) and (5).*' While the 
authors would indicate that the use of parallel forms of 
tests for practice is at the fraudulent end of the cor*tinuum, 
the use of the parallel form of a test as an alternate 
measuring instniment remains consistent with the psrallfcl 
testing design. 

Beyond the issues of test preparation activities and 
their appropriateness, lies the issue of test familiarity. 
In a second inquiry into the issues raised by Dr. Cannell, 
Gary Phillips (1990) restated the potential impact of test 
familiarity. "As the test is repeatedly used, the school 
system becomes more and more familiar with the test content 
and format. Again, tnis gives the test user an advantage not 
shared by the norming sample." In a related discussion, 
Shepard (1990) addresses the test familiarity question. 
Shepard speculates that "test familiarity might allow 
teachers to improve the performance of their students 
innocently, without consciously deciding to cheat, by 
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xeroxing a copy of the test." The author indicates that a 
teacher who simply remembers the nature of a test item and 
adjusts his or her instruction towards that item could serve 
lio influence a child's percentile score by two to seven 
points. This phenomenon could easily, even innocently, occur 
when a partic'ilar test is used for several years in a school 
system. 

Background for the Study 

The alternate form scheduling approach has not been 
utilized in the Cleveland City School District since the 
adoption of the California Achievement Test in 1987. 
Annually, the reading subtest (form E) of the CAT has been 
used at all grade levels within the district. Each spring, 
all test eligible pupils in the district are selected to take 
the appropriate level of the CAT reading test. In light of 
this factor, a question of test familiarity was raised 
concerning its potential impact upon achievement measures. 

During the 1989-90 academic year, the District initiated 
a separate testing program at grades four, six, and eight in 
order to conform to z> State achievement/ability testing 
program. Consequently, form F of the CAT battery was 
selected for this administration in the fall of 1989. The 
availability of these s«Jores enabled the district to address 
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the questions stated above. 



Purpose of the Study 

The purpose of this study was to explore the impact 
of evidence of test familiarity upon obtained achievement 
scores in reading. 

Specifically, the following questions were addressed. 

1. Did significant differences exist between CAT-E 
reading achievement scores and CAT-F reading 
achievement scores for students at various grade 
levels? 

2. Did student reading achievement trends differ 
significantly when measured by CAT-E or CAT-F tests? 

Methodology 

Sample 

The sample for this study included all students who were 
tested (grades four, six, and eight) in the program in fall, 
1989. This group formed the basis for the derivation of a 
cohort at each grade level. Readint, test rcores for these 
students were then selected from spring, 1988, spring, 1989 
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and spring, 1990. Those students who bad available scores 
for all four data points were included in the sample. The 
first cohort contained 4,362 nenbers who were fourth graders 
in 1989-90. The second cohort contained 3,768 members who 
were sixth graders in 1989-90 and the third cohort con^-uined 
2770 students who were in the eighth grade in 1989-90. 
Between three and five percent of each cohort membership 
repeated a grade during the period included in the study; 
these students took the same level and form of the reading 
test during their retention year. 

Table 1. 

Grade Levels of Cohort Members 



Cohort springes SPringB9 Fall89 SPring90 n 
GRO GRD GRD GRD 



1 2 3 4 4 4362 

2 4 5 6 6 3768 

3 6 7 8 8 2770 



Data Analysis and Results 



o 

ERIC 



9 



8 



Normal curve equivalent reading scores were included for 
data analyses to address the first question. Specifically, 
each cohort group's mean reading comprehension NCE was 
computed for the four testing sessions included in the study. 
A set of correlated t-tests between test sessions one and 
two, two and three, three and four, and two and four were 
conducted. Normal Curve Equivalent scores were chosen since 
the nationally norm referenced test was administered on level 
and scored during the empirical norming period for all 
testing sessions. Due to these factors, the mean NCE score 
for each cohort would be expected to remain consistent from 
one session to the next. Correlation coefficients would also 
be expected to remain high for all pairs. 

The results illustrated in the following tables are 
discussed below. 

The first cohort (grade four, 1989-90) evidenced an 
inconsistent achievement pattern from spring, 1988 to spring, 
1989 to spring, 1990 on the CAT form E. Mean NCE's increased 
by three points then dropped four points over the three 
spring measures. All means differed significantly (p< .01). 
However, the fall, 1989 measure -Jbtained by administering 
form F of the CAT was consistent with the spring, 1990 form E 
measure and indicated no significant difference with that 
measure. 
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COHORT I 



- - T-TCSTS FOR PAIRED SAMPLES 



VARIABLE NUMBER J^$^5^5S ^^6SSSS^ 

OF CASES MEAN DEVIATION ERROR 

SMHCE SPRINO S8 OBTAINED SCORE 

4362 46.6662 17.809 .270 

4362 49.6130 16.355 .248 

S89NCE SPRING 89 OCTAINEO SCORE 

(DIFFERENCE) STANDARD STANDARD I 2-TAIL I T 5E0REES OF 2-TAIL 

MEAN DEVIATION ERROK ^.52!!:-™!:-^-™-! !5!!BS!! 

-2^9269 I5T7OI T238 "l .580 .000 I -12.31 ^^561 .000 

VARIABLE NUMBER JI}?555S ^'^KSiS'^ 

OF CASES MEAN DEVIATION ERROR _ 

S89NCE SPRINO 89 OBTAINED SCORE ,„ 

4362 49.6130 16.355 .248 

4362 44.7354 15.893 .241 

F89NCE FALL 89 OBTAINED SCORE 

(DIFFERENCE) STANDARD STANDARD I ^^^^ 2-TAIL I T PEg5||5 2-TAIL 

MEAN DEVIATION ERROR I CORR. PROB. I VALUE ™!- 

4^8796 iT.llz ^207 1 .640 .000 I 23.56 4561 .000 

VARIABLE NUMBER STANDARD STANDARD 

OF CASES MEAN DEVIATION ^ ^ERROR 

S89iicE SPRINO 89 OBTAINED SCORE 

4362 49.6130 16.355 .248 

4562 45.1729 16.309 .2'i7 

S90NCE SPRING 90 DETAINED SCORE 

(DIFFERENCE) STANDARD STANDARD I 2-TAIL I T ^^EOREES OF 2-TAU 
MEAN DEVIATION ERROR I CORR. PROB. I ^VALUE f5^5£?^ t^lli 

474402 13T876 " .210 I .639 .000 I 21.13 4361 .000 

VARIABLE NUMBER STANDARD STANDARD 

OF CASES MEAN DEVIATION ERROR 

F89NCE FALL 89 OBTAINED SCORE 

4362 44.7334 15.893 .241 

4362 45.1729 16.309 .247 

S90NCE SPRING 90 OBTAINED SCORE 

(DIFFERENCE) STANDARD STANDARD I 2-TAIL I T DEGREES OF 2-TAIL 
MEAN DEVIATION ERROR I CORR. PROB. I VALUE FREEDOM PROB. 
— — — ♦ — -+ — — - 

-.4395 12.080 .185 I .719 .000 I -2.40 4361 .016 



Fiqyrg I* Achievement score comparisons for Cohort 1. 
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F89GR0< 6 COHORT 2 10 

T-TESTS FOR PAIRED SAMPLES 



VARIABLE NUMBER STANDARD STANDARD 

OF CASES HEAN DEVIATION ERROR 

SBBNCe" 'spring 88 OBTAINED SCORE 

3768 47.3992 U.786 .241 

5768 47.6746 14.683 .239 

S89NCE SPRING 89 OBTAINED SCORE 

(DIFFERENCE) STANDARD STANDARD I 2-TAU 2 T DEGREES OF 2-TAIL 

MEAN DEVIATION ERROR I CORR. PROB. I VALUE ^FREEDOM PROB. 

1^^724 ' ' .175 I .755 .000 I -1.58 S767 All 



VARIABLE NUMBER STANDARD STANDARD 

OF CASES MEAN DEVIATION ERROR 

S89NCE SPRING 89 OBTAINED SCORE 

5768 47.6746 14.685 .259 

3768 55.5587 15.840 .258 

F89NCE FALL 89 OBTAINED SCORE 

(DIFFERENCE) STANDARD STANDARD I 2-TAlL I T DEGREES OF 2-TAIL 
MEAN DEVIATION ERROR I CORR. PROB. I VALUE FREEDOM PROB. 
+ — 4- — 

-5.86<il 11.116 .181 I .757 .000 I -52.38 3767 .000 

VARIABLE NUMBER STANDARD STANDARD 

OF CASES MEAN DEVIATION ERROR 

S89NCE SPRING 89 OBTAINED SCORE 

5768 47.6746 14.685 .259 

376S 48.3129 16.604 .270 

S90NCE SPRING 90 OBTAINED SCORE 

(DIFFERENCE) STANDARD STANDARD I 2-TAU I T DEGREES OF 2-TAIL 

MEMN DEVIATION ERROR I CORR. PROB. I VALUE FREEDOM PROB. 

" -76583 ll'^n .192 ' I .725 .000 I -5.35 3767 .001 

VARIABLE NUMBER STANDARD STANDARD 

OF CASES MEAN DEVIATION ERROR 

F89NCE FALL 89 OBTAINED SCORE 

5768 53.5587 15.840 .258 

5768 48.5129 16.604 .270 

S90NCE SPRING 90 OBTAINED SCORE 

(DIFFERENCE) STANDARD STANDARD 1 2-TAU I T DEGREES OF 2-TAIL 

^ ME^N DEVIATION ERROR I CORR. PROB. I VALUE FREEDOM PROB. 

5.2258 11.554 .188 I .747 .000 I 27.76"'"3767 Tooo" 



Achievement score comparisons for Cohort 2. 
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COHORT 5 

T-TESTS FOR PAIREO SAMPLES - - 



VARIABLE NUMBER STANDARD ."ANDARD 

OF CASES MEAN DEVIATION ERROR 

SSBNCE SPRING 88 OBTAINED SCORE 

2770 50.7614 11 .410 .295 

2770 46.tfS54 IS. 865 .26S 

S89NCE SPRING 89 OBTAINED SCORE 

(DIFFERENCE) STANDARD STANDARD I 2-TAIL I T DEGREES OF 2-TAIL 
MEAN DEVIATION ERROR I CORK. PROB. I VALUE FREEDOM PROB. 
........... — — f-- ♦ 

5.9260 10.405 .198 I .752 .000 I 19.86 2769 .000 

VARIABLE NUMBER STANDARD STANDARD 

OF CASES MEAN DEVIATION ERROR 

S89NCE SPRING 89 OBTAINED SCORE 

2770 46.8554 15.865 .265 

2770 44.6585 17.515 .555 

F89NCE FALL ^9 OBTAINED SCORE 

C DIFFERENCE) STANDARD STANDARD I 2-TAIL I T DEGREES OF 2-TAU 
MEAN DEVIATION ERROR T CORR. PROB. I VALUE FREEDOM PROB. 
. — + — 

2.1971 11.610 .221 I .750 .000 I 9.96 2769 .000 

VARIABLE NUMBER STANDARD STANDARD 

OF CASES MEAN DEVIATION ERROR 

S89NCE f.PRING 89 OFTAIriED SCORF 

2770 46.8554 15.865 .262 

2770 45.0450 17.250 .528 

S90NCE SPRING 90 0I*TAINF.D SCORE 

(DIFFERENCE) STANDARD STANDARD I 2-TAIL I T DEGREES OF 2-TAIL 
MEAN DEVIATION ERROR I CORR. PROB. I VALUE FREEDOM PROB. 
♦ — — —4 

1.7924 12.758 .242 I .685 .000 I 7.41 2769 .000 

VARIABLE NUMBER STANDARD STANDARD 

OF CASES MEAN DEVIATION ERROR 

F89NCE FALL 89 OBTAINED SCORE 

277D 44.6585 17.515 .555 

2770 45.0450 17.250 .528 

S90NCE SPRING 90 OBTAINED SCORE 

(DIFFERENCE) STANDARD STANDARD I 2-TAIL I T DEGREES OF 2-TAU 

MEAN DEVIATION ERROR I CORR. PROB. I VALUE FREEDOM PROB. 

— — 4 — 4 

-.<i047 15.505 .257 I .698 .000 I -1.58 2769 .115 

Flq^rg 3f Achievement score comparisons for Cohort 3. 
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The seconu cohort (grade six, 1989-90) evidenced -a 
consistent pattern of achievement (means within one NCE 
point) over the three spring test sessions. Fall testing of 
this cohort evidenced a six NCK point (significant) rise from 
the previous form E session. 

Final?.y, the third cohort (grade eight, 1989-90) 
realized a four and two NCE point decline respectively (both 
significant) on three spring Form E measures. Fall (form F 
measures also showed the two point significant decline and a 
non-significant change to the subsequent spring measure. 

Although the assumption concerning consistency of the 
mean score over time was not demonstrated by these data, the 
consistency of fall (form F) to spring (form E) means amo- ■: 
the first and third cohort members illustrates little 
apparent impact of the use of the different form of the test. 

To address the second question, a regression 
discontinuity design was utilized to investigate the trends 
(implicit in the above analysis) in reading achievement 
evident for each cohort. Spring, 1988 NCE scores were 
plotted with spring, 1989 NCE scores to build a linear 
prediction equation for each of the cohorts. The equations 
were then used to calculate a predicted NCE score for each 
cohort member for the 1989 academic year. The predicted 
scores were then compared to the obtained fall (form F) and 
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spring (form £} scores to determine the differences. . .The 
illustrated t-tests and discussion follow. 

T:«e first cohort evidenced similar significant 
differences in measure between both forms of the cest and the 
predicted scores. In both comparisons, obtained scores were 
1.2 and 1.6 NCE points lower than predicted levels. 

The second cohort showed a wider (significant) 
difference between the two !aeasures and the predicted scores. 
Form F measured reading 6.3 NCE points higher than predicted 
while form E measured reading achievement 1.2 points higher 
than predicted. 

The third cohort, like the first, indicated lower 
obtained scores on both forms than predicted (6.5 and 6.1 
respectively. Again, the differences were both significant. 

With the exception of the second cohort, the alternate 
forms of the test indicate similar patterns of achievement 
(as might have been anticipated from the first analysis) . 

Conclusions 

The review of the achievement patterns over time of the 
three cohorts in this study indicated that in two of the 
three cases, the alternate form of the test yielded no 



ERIC 



F69GRI>t 



COHORT 1 

T-TESTS FOR PAIRED SAMPLES 



VARIABLE 



NUMBER 
OF CASES 



MEAN 



STANDARD 
DEVIATION 



STANDARD 
ERROR 



ERCNCE PREDICTED SCORE 

4562 46.3554 
4362 44.7SS4 

F89NCE FALL 89 OBTAINED SCORE 

(DIFFERENCE) STANDARD STANDARD 
MEAN DEVIATION ERROR 



10.S35 
15.893 



.157 
.241 



I 2-TAIL I 

I CORR. PROB. I 



T 

VALUE 



DEGREES OF 
FREEDOM 



2-TAIL 
PROB. 



1.6221 



12.208 



.185 



I .640 .000 I 8.78 



4361 



.000 



VARIABLE 



NUMBER 
OF CASE'i 



HEAN 



STANDARD 
DEVIATION 



STANDARD 
ERROR 



ERCNCE PREDICTED SCORE 

4362 46.3554 10.336 

4562 45.1729 16.309 

S90NCE SPRING 90 OBTAINED SCORE 



.157 
.247 



CDIFFERENCE) STANDARD STANDARD I 2-TAIL I T DEGREES OF 2-TAIL 
MEAN JfVf^JJJN ERROR I CORR. PROB. I VAIUE FREEDOM PROB. 

1.1826 12.544 .190 



I .639 .000 I 6.23 



4361 



.000 



riqwr^ 4 1 Predicted vewrsus Obtained Achievement scores for Cohort i. 
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Fe90RD. 6 COHORT 2 _ ^ ^^^^^^^ ^^^^^^ ^^^p.^S . . . 



VARIABLE NUMBER STANDARD STANDARD 

VARIABLE ^^^iJIi?? 

ERciicE PREDICTED SCORE 

S768 ^7.2792 ig-f*! 'Ill 

576a 53.5587 15.8*0 .258 

F8^rti:E FALL 89 OBT/INEO SCORE 



-r2595 I0T73S Tni l"".757 .000 I -55.80 S767 ,000 

VARIABLE NUMBER STANDARD STANDARD 

VARIABLE pNugjiij E!!iili2!! !!!?!.- 

ERci5cE PREDICTED SCORE 

3768 ^7.2792 10.865 .177 

3768 <i8.3129 U 60A .270 

S90NCE SPRING 90 OBTAINED SCORE 

(DIFFERENCE) STANDARD STANDARD I ^^^^ 2:TAIL I T ^'l^l^ 

MEAN DEVIATION ERROR I CORR^ ------ 

-r0337 llTiii'"^ ".ill l"'T72r "000"r""r51 5767 .000 



Figure 5. Predicted vewrsus Obtained Achievement sjores for Cohort 2. 
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COHORT S ^ _ x-TESTS FOR PAIRED SAMPLES - - - 



VARIA8LE NUMBER JJ.JWJSS 

OF CASES MEAN DEVIATION ERROR 

ERCNCE PREDICTED StORE 

2770 A<».6585 17.515 .355 

FS9NCE FALL £9 OBTAINED SCORE 

CDIFFERENCE) STANDARD STANDARD I 2-TAU I T DEGREES OF 2-TAIL 

MEAN DEVIATION ERROR X CORR.^PROB. I^_VJLUE [?5E?£- !52!l 

------ ll'lll ",111" I .750 .000 I 29.5-4 2769 .000 

VARIABLE NUMBER JT^^^5$SB ^"^JSSSS'' 

OF CASES MEAN DEVIATION ERROR 

ERCNCE PREDICTED SCORE ^ 

2770 51.15<i«i 11-591 .220 

2770 ^5.0^50 17.250 .328 

S90NCE SPRING 90 DETAINED SCORE 

(DIFFERENCE) STANDARD STANDARD I ^^^^ 2-TAIL I T DEGREES OF 2-TAIL 
MEAN DEVIATION ERROR I CORR. PROB. 1^_VALUE f?5!?5!? 

I'lill I2T573 "239 I .685 .000 1 25.58 2769 .000 



riquy^ $f Predicted vewrsus obtained Achievement scores for Cohort 3. 



18 



17 

significant differences than the subsequent administration of 
the test's other form. This fact was not evident among the 
second (sixth grade) cohort members in the study. These 
results among the first and third cohorts (as veil as the 
higher achievement measure obtained by the second cohort) 
refute the notion that repeated use of the same form of the 
test improves scores through '^practice effects.** 

Concerning reading achievement trends, again, two of the 
three cohorts (the first and third) demonstrated significant 
but comparable differences between the predicted and measured 
level of reading achievement. The middle cohort evidenced a 
considerably higher obtained than predicted reading level 
using this analysis. As above, the evidence does not support 
the contention that repeated use of a test contributes to 
improved scores. 

It appeared that alternative form test utilized during 
the same academic year provided comparable (or higher) 
reading achievement levels when utilized for the first time 
in the school district. 
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